Traffic control on an on-chip network

ABSTRACT

An apparatus, system, and method for controlling traffic on an on-chip network. Embodiments of the method comprise injecting a packet at a first rate into the on-chip network by a first node coupled to the on-chip network, receiving the packet at a second node coupled to the on-chip network, modifying a bit in the packet by the second node in response to determining that a rate at which packets are injected into the on-chip network should change, returning the packet with the bit modified to the first node by the second node, and changing the first rate by the first node in response to detecting that the bit in the packet was modified.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application is a U.S. National Phase Application under 35U.S.C. §371 of International Application No. PCT/US2011/067264, filedDec. 23, 2011, entitled TRAFFIC CONTROL ON AN ON-CHIP NETWORK.

FIELD

Embodiments of the invention relate generally to traffic on on-chipnetworks, and more particularly to controlling traffic on on-chipnetworks.

BACKGROUND

Components on a computer chip may be organized as an on-chip network ofnodes. For example, a multicore processor may be organized as an on-chipnetwork of cores. High traffic on on-chip networks (e.g., due to trafficbursts) can result in performance decreases, increased powerconsumption, and decreased quality of service.

BRIEF DESCRIPTION OF THE DRAWINGS

The following description includes discussion of figures havingillustrations given by way of example of implementations of embodimentsof the invention. The drawings should be understood by way of example,and not by way of limitation. As used herein, references to one or more“embodiments” are to be understood as describing a particular feature,structure, or characteristic included in at least one implementation ofthe invention. Thus, phrases such as “in one embodiment” appearingherein describe various embodiments and implementations of theinvention, and do not necessarily all refer to the same embodiment.However, they are also not necessarily mutually exclusive.

FIG. 1 is a block diagram showing an on-chip network that implementsdeflection by rerouting deflected packets to their destination, and anexemplary route taken by a packet on the network.

FIG. 2 is a block diagram of a packet with a “deflected” bit and a“throttle down” bit for use with routing algorithms to control trafficon an on-chip network according to one embodiment.

FIG. 3 is a block diagram of an on-chip network showing an exemplaryroute taken by a deflected packet that is returned to the source node bythe destination node according to one embodiment.

FIG. 4 is a block diagram of an on-chip network showing an exemplaryroute taken by a deflected packet that is returned to the source node byan intermediate node according to one embodiment.

FIG. 5 is a block diagram of a node coupled to an on-chip networkaccording to one embodiment.

FIG. 6 is a flow diagram of a process for controlling traffic on anon-chip network according to one embodiment.

FIG. 7 is a flow diagram of a process for handling a packet at anintermediate or a destination node according to one embodiment.

FIG. 8 is a flow diagram of a process for handling a packet that wasreturned to its source node according to one embodiment.

FIG. 9 is a block diagram of a computing system with a multicoreprocessor with cores organized as an on-chip network in whichembodiments of the invention may operate, be executed, integrated,and/or configured.

Descriptions of certain details and implementations follow, including adescription of the figures, which may depict some or all of theembodiments described below, as well as discussing other potentialembodiments or implementations of the inventive concepts presentedherein. An overview of embodiments of the invention is provided below,followed by a more detailed description with reference to the drawings.

DETAILED DESCRIPTION

Embodiments of the invention provide a method, apparatus, and system forcontrolling traffic on an on-chip network (e.g., a multicore processorwith cores organized as an on-chip network). Traffic on an on-chipnetwork could comprise, for example, packets (e.g., transaction packets)being sent from source nodes to destination nodes, some of which arerouted via intermediate nodes coupled to the on-chip network.

According to one embodiment, traffic on the on-chip network can becontrolled or influenced by using extra bits in the packet format androuting algorithms to be executed on nodes coupled to the network. Inone embodiment, the extra bits and routing algorithms enable packets tobe deflected back to their source nodes carrying information regardingchanging a packet-injection rate. In one such embodiment, thisinformation exchange via deflected packets between nodes coupled to anetwork provides for a mechanism for controlling traffic on on-chipnetworks.

On-chip networks may be implemented in a variety of ways, including, forexample, router-based mesh networks. A router-based mesh network is anetwork of nodes in which interconnected nodes relay packets to theirdestinations. In a router-based mesh network, each node and each packetsent from one node to another node on the on-chip network (i.e., from asource node to a destination node) have unique identifying information.To assist in relaying packets, buffering may be provided at both thedestination nodes and/or at the intermediate nodes to store packetsalong their way to the destination nodes. Some buffered networkimplementations may employ credits to notify sender-nodes when bufferspace on intermediate and/or destination nodes becomes available.

Although buffered, router-based mesh networks can have the benefit ofscalability, there can be drawbacks. For example, on-chip mesh networkshaving buffering resources can require significant chip area and power.One way to minimize buffering resources (and therefore require less chiparea and reduce power consumption) is to implement a network withdeflection routing. Networks using deflection routing allow the“bouncing” (i.e., deflection) of packets that are rejected atdestination nodes and/or at intermediate nodes. Packets may be rejecteddue to a variety of reasons including, for example, insufficientresources at a node for handling incoming packets. Typically, packetsthat are deflected may be rerouted to use different channels to reachtheir destinations. Deflection routing can permit a reduction in buffersizes, but can result in increased traffic on the on-chip network assome packets will be deflected rather than buffered.

Embodiments of the invention include a mechanism to reduce bouncingtraffic in a generic buffered router-based mesh network that employsdeflection at the endpoints and/or at the intermediate routers. In oneembodiment, one or more bits are included in the packet format thatenable deflecting nodes to indicate that a packet-injection rate shouldchange (e.g., a “throttle down” bit), and deflected packets are returnedto their source nodes. According to one embodiment, a source node, inresponse to detecting the bit set in a returned packet, adjusts itspacket-injection rate.

FIG. 1 is a block diagram 100 showing an on-chip network that implementsdeflection by rerouting deflected packets to their destination, and anexemplary route taken by a packet on the network. On-chip mesh network101 includes interconnected nodes 102 a-102 p. In this example, sourcenode 102 a injects a packet into the on-chip mesh network 101 to whichit is coupled; the packet's final target is destination node 1021, whichis also coupled to the on-chip network 101.

Initially, the packet was to travel down an initial route 104(represented by the solid line) from the source node 102 a to thedestination node 1021. However, in this example, the packet is deflectedby an intermediate node (e.g., node 102 c). Once the packet isdeflected, deflection routing may involve modifying the routingalgorithms of intermediate routers so that a packet that cannot berouted on an initial route (e.g., route 104) is routed on an alternativeroute (e.g., route 106, which is represented by the dashed line). Inthis example, the packet is deflected a second time by destination node1021. After being deflected at the destination node 1021, the packet cantake a route such as route 108 (represented by the dotted line) back tothe destination node.

FIG. 2 is a block diagram of one embodiment of a packet with additionalbits for use with routing algorithms to control traffic on an on-chipnetwork. In one embodiment, the packet 200 includes data 202, controlinformation 204, a “deflected” bit 206, and a “throttle down” bit 208.The control information 204 may include, for example, informationidentifying the packet 200, the packet's source node, and the packet'sdestination node. In one embodiment, the control information 204 furtherincludes information identifying the route the packet has taken from itssource node.

In one embodiment, the “deflected” bit 206 is modified by a deflectingnode (e.g., node 102 c or node 1021 in FIG. 1) to indicate that thepacket 200 was deflected. In one embodiment, the “throttle down” bit208, is set by a deflecting node if the deflecting node determines thatinjected traffic should be reduced. According to one embodiment, afterbeing deflected by an intermediate or destination node, instead of beingrerouted to its destination node, the packet 200 is returned to itssource node (e.g., node 102 a in FIG. 1). Once the packet arrives at itssource node, the source node changes its packet-injection rate accordingto the “throttle down” bit 208. For example, the source node can delayresending the packet 200 to the destination node if the “throttle down”bit 208 is set.

FIG. 3 is a block diagram 300 of an embodiment of an on-chip networkshowing an exemplary route taken by a deflected packet that is returnedto the source node by the destination node. According to one embodiment,nodes 302 a-302 p transmit packets having a format that includes a“throttle down” bit and a “deflected” bit (e.g., the packet 200 in FIG.2). In one embodiment, returning a deflected node to its source nodewith one or more bits modified to relay information can help controltraffic on the on-chip network.

For example, in one embodiment, a source node 302 a injects a packetinto the network 301 and the packet is routed to its destination node302 l via intermediate nodes 302 b, 302 c, 302 d, and 302 h (shown byroute 304 and represented by a solid line). At destination node 302 l,the destination node 302 l determines whether to deflect the packet. Asmentioned above, packets may be deflected for a number of reasons,including the destination node 302 l having insufficient space availablein its buffer.

In addition to deciding whether to deflect the packet, in one embodimentthe destination node 302 l determines whether a packet-injection rate onthe on-chip network should change. For example, in one embodiment thedestination node determines whether the rate at which the source node302 a injects packets should be decreased (i.e., throttled down). Thedetermination of whether a packet-injection rate should change can bebased on a number of factors. In one embodiment an algorithm based onthe monitoring of microarchitectural events and/or resource usage can beused to determine if the destination node is busy and requires thisaction. For example, counters (e.g., anti-starvation counters) can beused to signal when the number of rejected packets has exceeded athreshold. In another embodiment the determination of whether to changea packet-injection rate is based on the fill level of some resources inthe node (e.g., a buffer).

Once the destination node 302 l determines whether a packet-injectionrate should be changed, the destination node 302 l can modify one ormore bits in the packet to communicate the desired packet-injection ratechange. For example, in one embodiment, the destination node 302 l canset a “throttle down” bit in the packet returned to the source node 302a. In one embodiment, the destination node 302 l can modify one or moreadditional bits to communicate whether the packet was deflected (e.g., a“deflected” bit). In one such embodiment, the “deflected” bit cancommunicate additional information to its source node and/or othernodes. For example, using a “deflected” bit in addition to a “throttledown” bit provides information about situations in which a packet isdeflected but where it is determined that no throttling down is needed(and therefore the deflecting node does not set a “throttle down” bit).

In the example in FIG. 3, the destination node 302 l deflects thepacket. In one embodiment, instead of the deflected packet being routedback to the destination node 302 l (like the deflected packet referredto with respect to FIG. 1), the packet is returned to its source node302 a. The deflected packet's route back to its source node 302 a isshown by route 306 (represented by a dashed line).

In one embodiment, once the source node 302 a receives the deflectedpacket, the source node 302 a adjusts its packet injection rateaccording to the information in the deflected packet (e.g., the“throttle down” bit) and resends the packet to the destination node 302l. Adjusting the source node's packet injection rate may include, forexample, waiting for a number of cycles before resending the packet.

According to one embodiment, by creating this communication loop betweenthe source node 302 a and the destination node 302 l, it is possible forthe destination node 302 l to attach extra information to the returningpacket (e.g., a request to throttle down the sender's packet injectionrate), and reduce high rates of deflection on the on-chip network.

FIG. 4 is a block diagram of an embodiment of an on-chip network showingan exemplary route taken by a deflected packet that is returned to thesource node by an intermediate node.

Similar to the embodiment illustrated in FIG. 3, a source node 302 ainjects a packet into the network 301 targeting destination node 302 l(as demonstrated by route 404, represented by a solid line to thedestination node 302 l). However, in this embodiment an intermediatenode 302 c deflects the packet back to source node 302 a (demonstratedby route 406, represented by a dashed line).

In one such embodiment, the intermediate deflecting node determineswhether to deflect the packet and whether a packet injection rate (e.g.,the source node's packet injection rate) should change. Similarly to thedeterminations by the destination node 302 l described in reference toFIG. 3, an intermediate node 302 c can modify one or more bits in thepacket (e.g., to indicate that the source node 302 a should be“throttled down”) and send the packet back to the source node 302 ausing the same algorithm(s). In one embodiment employing deflection atintermediate nodes, the implementation includes mechanisms for avoiding“nested bouncing.” For example, such mechanisms could include logic toensure that when a destination node has deflected a packet, the packetis not deflected again at an intermediate node. One way to accomplishthis is through the use of a “deflected bit” in the packet, as describedabove with reference to FIG. 2.

The on-chip networks illustrated in FIGS. 1, 3, and 4 are exemplary andare not intended to be limiting. Embodiments of the invention areapplicable for other configurations of on-chip networks.

FIG. 5 is a block diagram of an embodiment of a node on an on-chipnetwork. In one embodiment, node 500 is a core of a multicore processorand includes processing unit 502 for executing instructions. In oneembodiment, node 500 further includes logic 504 to receive packets,logic 506 to transmit packets (i.e., inject packets into the network),switch logic 508 for determining a received packet's destination, logic510 to determine whether a packet-injection rate should be changed,logic 512 to perform configuration changes, buffer(s) 514 for holdingpackets, and register(s) 516. Register(s) 516 may include informationuniquely identifying node 500, and/or control information such as thenode's packet-injection rate.

According to one embodiment, logic 504 to receive packets may includelogic 518 to determine if a received packet should be deflected, andswitch logic 508 includes logic to determine a route of a packet to bedeflected. In one embodiment, switch logic 508 determines a deflectedpacket's route according to a specific policy. For example, a deflectedpacket can be sent via a pre-defined route (e.g., the same path by whichit came to the deflecting node) or through a different path using adifferent routing algorithm (e.g., by using global considerations suchas congestion). According to one embodiment, a deflected packet's routeis chosen according to a policy such as round-robin or randomly. Around-robin policy may comprise choosing the next channel according toan order (every time a channel is to be chosen, the next channel inorder is selected). In case of deflecting at a destination node, in oneembodiment an algorithm to select the deflected packet's route selects aroute to avoid congestion around the destination node (e.g., to avoidtight loops around the destination node).

Before transmitting a packet that is to be deflected, logic 510determines whether a packet-injection rate should change. For example,according to one embodiment, logic 510 determines whether the deflectedpacket's source node's packet-injection rate should be throttled down.If logic 510 determines that a packet-injection rate should change, oneor more bits in the packet are modified to communicate that decision toanother node (e.g., the source node).

FIG. 6 is a flow diagram 600 of an embodiment of a process forcontrolling traffic on an on-chip network. According to one embodiment,at block 604, a packet is injected into the on-chip network at a firstrate by a first node. In the case where the packet originates at thefirst node, the first node is the packet's source node. At block 606,the packet is received by a second node, which could be the packet'sdestination node or an intermediate node. At block 608, the second nodedetermines whether a packet-injection rate should be changed (e.g.,determining whether the packet's source node should be throttled down).In one embodiment, the determination of whether a packet-injection rateshould be changed only occurs after the node determines that the packetwill be deflected.

At block 610, the second node modifies a bit in the packet to indicatethat a packet-injection rate should change. For example, the second nodecan set a bit in the packet to indicate that the second node is busy andthat traffic should be throttled down.

At block 612, the second node returns the packet to its source node(i.e., the first node). The packet may be returned using a pathdetermined according to a specific policy (e.g., the path the packettook from the first node to the second node, or a random path back tothe first node).

At block 614, the first node changes the first rate (e.g., the firstnode's packet-injection rate) in response to detecting that the bit inthe packet was modified. For example, the source node can decrease itspacket-injection rate, and/or delay resending the deflected packet toits destination. In one embodiment, the method describe in flow diagram600 provides a means by which traffic on the on-chip network iscontrolled by changing the first node's packet-injection rate.

FIG. 7 is a flow diagram of an embodiment of a process for handling apacket at an intermediate or destination node. Flow diagram 700 beginsat block 702, where a packet reaches an intermediate or destinationnode. At decision block 704, the node determines whether to deflect thepacket. If the packet is not deflected, the packet is processed by thenode at block 706. Processing the packet may include, for example,determining that the packet's destination is another node and forwardingthe packet to that node, or determining that the packet's destinationhas been reached. If the packet's destination has been reached, the nodemay then execute instructions according to the packet's contents.

If the packet is to be deflected, the node also determines whether apacket-injection rate should be changed. For example, in one embodimentthe node determines whether anti-starvation counter(s) exceed apredetermined threshold at decision block 708. If the anti-starvationcounter(s) exceed the predetermined threshold, a “throttle down” bit inthe packet is set at block 710, and the packet is sent back to itssource node at block 712. According to one embodiment, if the counter(s)do not exceed the threshold, the packet is sent back to its originwithout setting the “throttle down” bit in the packet at block 712.

FIG. 8 is a flow diagram of an embodiment of a process for handling apacket that was returned to its source node (e.g., as in block 712 ofFIG. 7). Flow diagram 900 begins at block 802 where a packet reaches itssource node (i.e., the node at which the packet originated). At decisionblock 804, the source node checks if a “throttle down” bit in the packetis set. If the “throttle down” bit is not set, the source node resendsthe packet to its destination according to its policy at block 808. Ifthe “throttle down” bit is set, the source node modifies itspacket-injection rate policy at block 806 and then resends the packetaccording to the policy at block 808. Different packet-injection ratepolicies may be used to determine when to resend the packet. Forexample, the packet can be re-sent immediately or after some time delay.

FIG. 9 is a block diagram of an embodiment of a computing system with amulticore processor with cores organized as an on-chip network in whichembodiments of the invention may operate, be executed, integrated,and/or configured. System 900 represents a computing device, and can bea laptop computer, a desktop computer, a server, a gaming orentertainment control system, a scanner, copier, printer, a tablet, orother electronic device. System 900 includes processor 920, whichprovides processing, operation management, and execution of instructionsfor system 900. Processor 920 can include any type of processinghardware having multiple processor cores 921 a-921 n to provideprocessing for system 900. Processor cores 921 a-921 n are organized asan interconnected on-chip network (e.g., as an on-chip router-based meshnetwork). Processor cores 921 a-921 n include logic to enable deflectionand routing of packets back to the packet's source node to reducebouncing traffic (i.e., traffic resulting from deflections). Embodimentsof the invention as described above may be implemented in system 900 viahardware, firmware, and/or software.

Memory 930 represents the main memory of system 900, and providestemporary storage for code to be executed by processor 920, or datavalues to be used in executing a routine. Memory 930 may include one ormore memory devices such as read-only memory (ROM), flash memory, one ormore varieties of random access memory (RAM), or other memory devices,or a combination of such devices. Memory 930 stores and hosts, amongother things, operating system (OS) 936 to provide a software platformfor execution of instructions in system 900. OS 936 is executed byprocessor 920.

Processor 920 and memory 930 are coupled to bus/bus system 910. Bus 910is an abstraction that represents any one or more separate physicalbuses, communication lines/interfaces, and/or point-to-pointconnections, connected by appropriate bridges, adapters, and/orcontrollers. Therefore, bus 910 can include, for example, one or more ofa system bus, a Peripheral Component Interconnect (PCI) bus, aHyperTransport or industry standard architecture (ISA) bus, a smallcomputer system interface (SCSI) bus, a universal serial bus (USB), oran Institute of Electrical and Electronics Engineers (IEEE) standard1394 bus (commonly referred to as “Firewire”). The buses of bus 910 canalso correspond to interfaces in network interface 950.

In one embodiment, bus 910 includes a data bus that is a data bus overwhich processor 930 can read values from memory 930. The additional lineshown linking processor 920 to memory subsystem 930 represents a commandbus over which processor 920 provides commands and addresses to accessmemory 930.

System 900 also includes one or more input/output (I/O) interface(s)940, network interface 950, one or more internal mass storage device(s)960, and peripheral interface 970 coupled to bus 910. I/O interface 940can include one or more interface components through which a userinteracts with system 900 (e.g., video, audio, and/or alphanumericinterfacing). Network interface 950 provides system 900 the ability tocommunicate with remote devices (e.g., servers, other computing devices)over one or more networks. Network interface 950 can include an Ethernetadapter, wireless interconnection components, USB (universal serialbus), or other wired or wireless standards-based or proprietaryinterfaces.

Storage 960 can be or include any conventional medium for storing datain a nonvolatile manner, such as one or more magnetic, solid state, oroptical based disks, or a combination. Storage 960 may hold code orinstructions and data in a persistent state (i.e., the value is retaineddespite interruption of power to system 900). Storage 960 may include anon-transitory machine-readable or computer readable storage medium onwhich is stored instructions (e.g., software and/or firmware) embodyingany one or more of the methodologies or functions described herein.

Peripheral interface 970 can include any hardware interface notspecifically mentioned above. Peripherals refer generally to devicesthat connect dependently to system 900. A dependent connection is onewhere system 900 provides the software and/or hardware platform on whichoperation executes, and with which a user interacts. Besides what isdescribed herein, various modifications can be made to the disclosedembodiments and implementations of the invention without departing fromtheir scope. Therefore, the illustrations and examples herein should beconstrued in an illustrative, and not a restrictive sense. Any of thedisclosed embodiments may be used alone or together with one another inany combination. Although various embodiments may have been partiallymotivated by deficiencies with conventional techniques and approaches,some of which are described or alluded to within the specification, theembodiments need not necessarily address or solve any of thesedeficiencies, but rather, may address only some of the deficiencies,address none of the deficiencies, or be directed toward differentdeficiencies and problems which are not directly discussed. The scope ofthe invention should be measured solely by reference to the claims thatfollow.

What is claimed is:
 1. A method of controlling traffic on an on-chipnetwork, the method comprising: injecting a packet at a first rate intothe on-chip network by a first node coupled to the on-chip network;receiving the packet at a second node coupled to the on-chip network;determining that the received packet is to be deflected and not bufferedby the second node; modifying a first bit in the received packet by thesecond node in response to determining that a rate at which packets areinjected into the on-chip network should change and a second bitdifferent than the first bit in the received packet to indicate that thereceived packet was deflected; returning the packet with the first andsecond bits modified to the first node by the second node; and changingthe first rate by the first node in response to detecting that the firstand second bits in the packet were modified.
 2. The method of claim 1,wherein determining by the second node that a rate at which packets areinjected into the on-chip network should change comprises determiningthat the first rate should be reduced.
 3. The method of claim 2, whereindetermining that the first rate should be reduced comprises determiningthat the second node lacks resources to handle the packet at the timethe second node receives the packet.
 4. The method of claim 2, whereindetermining that the first rate should be reduced comprises determiningthat a number of packets deflected by the second node exceeds athreshold number.
 5. The method of claim 1, wherein the second node is adestination node for the packet.
 6. The method of claim 1, wherein thesecond node is an intermediate node to forward the packet to adestination node.
 7. The method of claim 1, wherein returning the packetto the first node comprises transmitting the packet to the first nodevia a pre-defined route.
 8. The method of claim 7, wherein thepre-defined route is a route the packet took from the first node to thesecond node.
 9. The method of claim 1, wherein returning the packet tothe first node comprises transmitting the packet to the first node via arandom route.
 10. The method of claim 1, wherein the on-chip network isa buffered network.
 11. A non-transitory computer-readable storagemedium having content stored thereon, which when executed causes one ormore processors having nodes organized as an on-chip buffered networkto: inject a packet at a first rate into the on-chip network by a firstnode coupled to the on-chip network; receive the packet at a second nodecoupled to the on-chip network; determine that the received packet is tobe deflected and not buffered by the second node; modify a first bit inthe received packet by the second node in response to determining that arate at which packets are injected into the on-chip network shouldchange and a second bit different than the first bit in the receivedpacket to indicate that the received packet was deflected; return thepacket with the first and second bits modified to the first node by thesecond node; and change the first rate by the first node in response todetecting that the first and second bits in the packet were modified.